Method and Apparatus for Better Web Ad Matching by Combining Relevance with Consumer Click Feedback

ABSTRACT

A method and apparatus are provided for better web ad matching by combining relevance with consumer click feedback. In one example, the method includes receiving a query page, extracting features from the query page, re-weighting the query page, evaluating the query page in light of each ad in order to score each ad and pick substantially best ad matches of the indexed ads, and returning the substantially best ad matches to the consumer computer.

FIELD OF THE INVENTION

The present invention relates to providing better web ads. Moreparticularly, the present invention relates to providing better web adsby matching words on query pages with words on clicked web ads.

BACKGROUND OF THE INVENTION

Web advertising provides financial support for a large portion oftoday's Internet ecosystem, catering to a diverse set of websites, suchas blogs, news, reviews, etc. Spurred by the tremendous growth intraffic in terms of volume, number of consumers, consumer engagement,content diversity, the last few years from 2008 have seen a tremendousgrowth in spending on web advertising.

A major part of the advertising on the web falls into the category oftextual ads, which are typically short textual messages usually markedas “sponsored links” or similar. There are two main types of textual adson the web today:

-   -   1. Sponsored search (i.e., paid search) advertising places ads        on the result pages from a web search engine based on the search        query. All major current web search engines support such ads and        act simultaneously as a search engine and an ad agency.    -   2. Contextual advertising (i.e., Context Match) advertising        places ads within the content of a generic, third-party web        page. There usually is a commercial intermediary, called an        ad-network, in charge of optimizing the ad selection with the        twin goals of increasing revenue (shared between publisher and        ad-network) and improving consumer experience. Here, the main        players are the major search engines; however, there are also        many smaller players.

While the methods proposed in this paper could be adapted for bothsponsored search sponsored search and contextual advertising, therelevant background is primarily contextual advertising.

Studies have shown that displaying ads that are closely related to thecontent of the page provide a better consumer experience and increasethe probability of clicks. This intuition is analogous to that inconventional publishing, where there are very successful magazines(e.g., Vogue) where a majority of the content is topical advertising(e.g., fashion, in the case of Vogue). Thus, estimating the relevance ofan ad to a page is critical in serving ads at run-time.

Previously, published approaches estimated the relevance based onco-occurrence of the same words or phrases within the ad and within thepage. The model used in this body of work is to translate the ad searchinto a similarity search in a vector space. Each ad is represented as avector of features, as for example, unigrams, phrases and classes. Thepage is also translated to a vector in the same space as the ads. Thesearch for the substantially best ads is now translated into finding thead vectors that are closest to the page vector. To make the searchefficient and scalable to hundreds of millions of ads and billions ofrequests per day, an ad system can use an inverted index and anefficient similarity search algorithm. A drawback of this method is thatit relies on a-priori information and does not use the feedback (aposteriori) information that is collected in the form of ad impressions(displays) and clicks.

Another line of work uses click data to produce a CTR (click throughrate) estimate for an ad, independent of the page (or query page, in thesponsored search scenario). The CTR is estimated based on featuresextracted from the ads that are then used in a learning framework tobuild models for estimation of the CTR of unseen ads. In this approach,the assumption is that the ad system selects the ads by a deterministicmethod—by matching the bid phrase to a phrase from the page (or thequery page in sponsored search). Accordingly, to select the mostclickable ads, the ad system only needs to estimate the CTR on the adswith the matching bid phrase. This simplifying assumption of thematching process is an obvious drawback of these approaches. Anotherdrawback is that these methods do not account for differential clickprobabilities on different pages: If some pages in the corpus attract anaudience that clicks on ads significantly more than average, then thelearning of feature weights for ads will be biased towards ads that were(only by circumstance) shown on such pages.

SUMMARY OF THE INVENTION

What is needed is an improved method having features for addressing theproblems mentioned above and new features not yet discussed. Broadlyspeaking, the present invention fills these needs by providing a methodand apparatus for providing better web ad matching by combiningrelevance with consumer click feedback. It should be appreciated thatthe present invention can be implemented in numerous ways, including asa method, a process, an apparatus, a system or a device. Inventiveembodiments of the present invention are summarized below.

In one embodiment, a method is provided for comparing query pages toindexed ads in order to provide better web ad matching. The methodcomprises receiving a query page, extracting features from the querypage, re-weighting the query page, evaluating the query page in light ofeach ad in order to score each ad and pick substantially best ad matchesof the indexed ads, and returning the substantially best ad matches tothe consumer computer.

In another embodiment, a method is provided for indexing ads in order toprovide better web ad matching. The method comprises receiving ads thatwere clicked at a consumer computer, extracting ad features from theads, sorting the ads according to ad identification to provide a datafile, and inverting the data file to sort the data file according tofeature identification, wherein sorting the ads includes computing astatic score for each ad using parameters learnt using logisticregression on some training data.

In still another embodiment, an apparatus is provided for comparingquery pages to indexed ads in order to provide better web ad matching,wherein the apparatus is configured to receive a query page. Theapparatus comprises a page feature extraction device configured toextract features from the query page, a page feature re-weighting deviceconfigured to re-weight the query page, a page evaluation deviceconfigured to evaluate the query page in light of each ad in order toscore each ad and pick to obtain substantially the best ad matches ofthe indexed ads, wherein the apparatus is configured to return thesubstantially best ad matches to the consumer computer.

In yet another embodiment, an apparatus is provided for indexing ads inorder to provide better web ad matching, wherein the apparatus isconfigured to receive ads that were clicked at a consumer computer. Theapparatus comprises an ad feature extraction device configured toextract ad features from the ads, an ad identification assignment deviceconfigured to sort the ads according to ad identification to provide adata file, and an ad inversion sort device configured to invert the datafile to sort the data file according to feature identification, whereinthe apparatus is further configured to sort the ads by computing astatic score for each ad using parameters learnt using logisticregression on some training data.

In still yet another embodiment, a computer readable medium is providedcarrying one or more instructions for comparing query pages to indexedads in order to provide better web ad matching. The one or moreinstructions, when executed by one or more processors, cause the one ormore processors to perform the steps of receiving a query page,extracting features from the query page, re-weighting the query page,evaluating the query page to obtain substantially best ad matches of theindexed ads, and calculating a final score for each ad in order to picksubstantially best ad matches.

The invention encompasses other embodiments configured as set forthabove and with other features and alternatives.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be readily understood by the followingdetailed description in conjunction with the accompanying drawings. Tofacilitate this description, like reference numerals designate likestructural elements.

FIG. 1 is a block diagram of a system for providing better web admatching by combining relevance with consumer click feedback, inaccordance with an embodiment of the present invention;

FIG. 2 is a schematic diagram of a system for providing better web admatching by combining relevance with consumer click feedback, inaccordance with an embodiment of the present invention;

FIG. 3 is a flowchart of a method of indexing ads in order to providebetter web ad matching, in accordance with an embodiment of the presentinvention; and

FIG. 4 is a flowchart of a method for comparing query pages to indexedads in order to provide better web ad matching, in accordance with anembodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

An invention for a method and apparatus for provided better web admatching by combining relevance with consumer click feedback isdisclosed. Numerous specific details are set forth in order to provide athorough understanding of the present invention. It will be understood,however, to one skilled in the art, that the present invention may bepracticed with other specific details.

General Overview

FIG. 1 is a block diagram of a system 100 for providing better web admatching by combining relevance with consumer click feedback, inaccordance with an embodiment of the present invention. A device of thepresent invention is hardware, software or a combination thereof. Adevice may sometimes be referred to as an apparatus. Each device isconfigured to carry out one or more steps of the method of providingbetter web ad matching by combining relevance with consumer clickfeedback.

The network 102 couples together a front end server 104, a consumercomputer 106, an ad server 110, an ads database 112, a click feedbackdevice 114, an ads index database 122 and a relevance device 124. Thenetwork 102 may be any combination of networks, including withoutlimitation the Internet, a local area network, a wide area network, awireless network and a cellular network. The click feedback device 114includes without limitation an ad feature extraction device 116, an adidentification assignment device 118, an ad sort device 120 and an adindexing device 121. The relevance device 124 includes withoutlimitation a page feature extraction device 126, a page featurere-weighting device 128, a page evaluation device 130 and a clickprobability device 132.

Alternatively, one apparatus may contain two or more devices of thesystem 100. For example, one apparatus may contain two or more of thedevices that include, for example, the front end server 104, the clickfeedback device 114, the ads index database 122 and the relevance device124.

The system 100 is based on logistic regression, a popular technique instatistics and machine learning. The regression enables the system 100to combine click feedback and semantic information available from bothpages and ads to determine relevancy. This system 100 is more generalthan a pure relevance based approach that does not use click feedback inany form. Indeed, experiments performed with the system 100 convincinglydemonstrate the usefulness of using click feedback to find more relevantads. There has been prior work that involves using regression models fordetermining relevant ads. While it has a similar flavor of the presentsystem 100, only ad-specific features are learnt in such prior art, andad specific features are only a subset of the features that the system100 utilizes. In particular, in addition to page and ad specificfeatures, the system 100 learns features that capture interactionsbetween pages and ads. Furthermore, the system 100 combines word-basedfeatures with traditional relevance measures to enhance matchingrelevant ads to pages.

The models of the system 100 are more granular and can incorporatelarger number of features. Such incorporation reduces bias in CTRestimates and leads to better performance. However, reduced bias comesat the price of increased variance, which can become a serious problemif the models become too granular and start over-fitting the trainingdata.

To balance these two issues, the system 100 utilizes a two-prongedstrategy. First, the system 100 uses a relatively large but speciallyselected set of features, where the selection mechanism ensures that thefeatures have reasonable support. The system 100 also provides amechanism based on prior probabilities to down-weight features that aretoo sparse.

The second strategy the system 100 uses to prevent over-fitting is forthe system 100 to train its models on an extremely large corpus (e.g.,billions of records, several thousand features), which automaticallyincreases the support of a large number of features. Fortunately, datais plentiful especially for big ad-networks that serve a large number ofpublishers and advertisers. However, increased training size poses adifficult computational challenge of scaling logistic regression to webscale data. The system 100 overcomes this difficulty by using anapproximation based on a “divide and conquer strategy”. In other words,the system 100 randomly splits its training corpus into several piecesand fits a separate logistic regression to each piece. The system 100obtains the final result by combining estimates from all the pieces.

The system 100 carries out a method that involves three broad steps—(a)feature extraction, (b) feature selection, and (c) coefficientestimation for features through a logistic regression. A detaileddescription of each of these steps is provided below.

Feature Extraction

The system 100 treats pages and ads as being composed of severalregions. For instance, a page is composed of page title, page metadata,page body, page URL etc. Similarly, an ad is composed of ad title, adbody etc. Within each region, the ad feature extraction device 116 andthe page feature extraction device 126 each extract a set ofwords/phrases after stop word removal. The system 100 associates a score(e.g., region specific tf, tf-idf) to each word that measures itsimportance in a given region. The score may be, for example, regionspecific tf (term frequency) or tf-idf (term frequency-inverse documentfrequency). For a given page/ad region combination, this model has threesets of features that are described below.

The first feature set is page region specific main effects. Web pagesare usually composed of multiple regions with different visibility andprominence. Accordingly, the impact of each region on the ad selectioncan vary. The system 100 learns the effect of each region separately.For a word w in page region p(r) with score t_(p(r)w), theregion-specific main effect is defined as

M _(p(r)w)=1(wεp(r))·t _(p(r)w).  Equation 1.

In other words, if the word is present in the page region p(r), thefeature contributes its score else it does not contribute. Thesefeatures provide an estimate of word popularity. These features are notuseful at the time of selecting relevant ads for a given page but helpin getting better estimates of other terms in the model after adjustingfor the effect of popular words on a page. For instance, if “camera”pages are popular in terms of CTRs and 90% of the corpus consists ofcamera pages, “camera” ads that were the ones mostly shown on camerapages would tend to become popular even on “soccer” pages whichconstitute only 1% of the total corpus. By incorporating page words inthe model, the system 100 adjusts for this effect and gets the correctmatching ads for “soccer” pages.

The second feature set is ad region specific main effects. Ads are alsocomposed of multiple regions, some visible to the consumer (title,abstract) and some used only in the ad selection (bid phrase, targetingattributes). As with the page regions, the ad regions can have adifferent impact on the ad selection. For a word w in ad region a(r)with score t_(a(r)w), this is defined as

M _(a(r)w)=1(wεa(r))·t _(a(r)w).  Equation 2.

Unlike page specific main effects, ad region specific main effects doplay an important role when selecting relevant ads for a given page andprovide more weight to popular ads.

The third feature set is interaction effects between page and adregions. For a word w₁ in page region p(r₁) and word w₂ in ad regiona(r₂) with score ƒ(t_(p)(r₁) w₁, t_(a)(r₂) w₂) for some function ƒ, thisis given as

I _(p(r) ₁ _()w) ₁ _(,a(r) ₂ _()w) ₂ _(,)=1(w ₁ εp(r ₁),w ₂ εa(r ₂))·ƒ(t_(p(r) ₁ _()w) ₁ ,t _(a(r) ₂ _()w) ₂ ).  Equation 3.

The system 100 confines itself to the case where w₁=w₂. In other words,the feature “fires” only if the same word occurs in both thecorresponding page and ad regions. However, one can generally considerco-occurrences of synonyms or related words. Examples of ƒ include theproduct function t_(p)(r₁) w₁×t_(a)(r₂) w₂, the geometric mean √{squareroot over (t_(p(r) ₁)w₁×t_(a(r) ₂)w₂)}{square root over (t_(p(r)₁)w₁×t_(a(r) ₂)w₂)} and so on. Interaction effects are importantcomponents of the system 100 and help in matching relevant ads to agiven page. For instance, occurrence of the word “camera” in the ad bodyis a strong indication of the ad being relevant for the page whose titlecontains the word “camera,” with the degree of relevance beingdetermined by the regression.

Feature Selection

For any given (page, ad) region combination, a large number of wordsoccur in the training data. Using them all as features might make thelogistic regression ill-conditioned and inflate variance of thecoefficient estimates. Accordingly, the system 100 takes recourse tovariable selection techniques which select a subset of important wordsto be used in its regression. Variable selection in the context ofregression is a well studied area with a rich literature. Stepwisebackward-forward automated variable selection algorithms are widely usedfor large scale applications, but these methods have drawbacks,especially when features are correlated. The general recommendation isto use as much domain knowledge as possible instead of using anautomated procedure to select relevant variables. However, in largescale settings as in the system 100, some level of automation isnecessary.

For reasons of scalability, the system 100 uses a two-stage approach. Inthe first stage, the system 100 conservatively prunes non-informativefeatures using simple measures that can be computed using only a fewpasses over the training corpus. In the second stage, the system 100fits a regression to all the selected features from the first stage butdown-weights them through a specially constructed prior that pools datafrom all the features. Meanwhile, the system 100 preferably picks thefeatures that are less sparse. The second state is discussed below inmore detail in the Approximate Logistic Regression section. The variableselection methods are discussed next.

The system 100 selects the variables using two methods. The first methodis based on clicks and views. The second method is based on relevancescores of words that are independent of any click feedback. In the firstapproach (data-based), the system 100 ranks words based on a measurethat quantifies the interaction between words occurring in the page andad regions. For a word w, the interaction measure is defined as

$\begin{matrix}{{I_{w} = \frac{{CTR}_{w}^{both}}{{CTR}_{w}^{page} \cdot {CTR}_{w}^{ad}}},.} & {{Equation}\mspace{14mu} 4}\end{matrix}$

where CTR_(w) ^(both) denotes the CTR when w occurred both on pageregion and ad region of an ad displayed on a page, and CTR_(w) ^(page)and CTR_(w) ^(ad) denote the marginal CTRs when w is shown on the pageand ad regions, respectively. Higher values of the ratio indicatestronger interaction being induced by the presence of the word w whichin turn should enhance the matching quality of ads to pages. A variationof the measure above may be tried with a square root of the denominator,which will likely yield with no significant impact.

In the second approach (relevance-based), words are ranked by computingthe average tf-idf scores across the entire page and ad corpus for therespective regions under consideration. Here, the system 100 may involvetwo measures: (a) Create a single corpus by treating page and ad regionsas documents and compute a single tf-idf average score for each word;and (b) Treat the page and ad regions as different corpora and use thegeometric mean of tf-idf scores computed separately from page and adregions for each word.

For both measures, the system 100 picks, for example, the top 1000 wordsand uses them in the logistic regression. To avoid noisy estimates ofCTRs in the ratio, the system 100 only considers words that are shownsimultaneously on ad and page regions at least 10 times and havenon-zero marginal probabilities. It turns out that the data-basedapproach gives better results for the same number of words.

Approximate Logistic Regression

Let y_(ij) denote the binary click outcome (1 for click, 0 for no click)when ad j is shown on page i. Assume y_(ij) has a Bernoulli distributionwith CTR p_(ij). In other words, the probability distribution of y_(ij)is given by P(y_(ij))=p_(ij) ^(y) ^(ij) (1−p_(ij))^(1-y) ^(ij) . Todetermine relevant ads for a given page i, the system 100 needs toestimate p_(ij)'s, with higher values indicating more relevant ads. Forads that are shown a large number of times on a page, the system 100 canestimate the CTR empirically by clicks per impression. However, forpurposes here, a large fraction of page-ad pairs have a small number ofimpressions. In fact, since the CTRs are typically low (0.1%-20% with asubstantial right skewness in the distribution), the number ofimpressions required to get precise empirical estimates are high. Forinstance, to estimate a 5% CTR, the system 100 needs 1,000 impressionsto be even 85% confident that the estimate is within 1% of the true CTR.Thus, the system 100 takes recourse to feature based models. In otherwords, p_(ij) is a function of features extracted from page and adregions as discussed above in the Feature Extraction section.

To allow for arbitrary real-valued coefficients for features, it isroutine to map p_(ij) onto the real line via a monotonically increasingfunction. The most widely used function is the logit which maps p_(ij)to logit(p_(ij))=log [p_(ij)/(1−p_(ij))]. Assume logit(p_(ij)) is alinear function of features representing the main effects andinteraction effects discussed in the Feature Extraction section. Forsimplicity, consider a single (page, ad) region combination (p(r₁),a(r₂)). The linear function in the logistic regression is given by

$\begin{matrix}{{{logit}\left( p_{ij} \right)} = {{{logit}\left( q_{ij} \right)} + {\sum\limits_{w}\; {\alpha_{w}M_{{p{(r_{1})}}w}}} + {\sum\limits_{w}\; {\beta_{w}M_{{a{(r_{2})}}w}}} + {\sum\limits_{w}\; {\delta_{w,r_{1},r_{2}}{I_{{{p{(r_{1})}}w},{{a{(r_{2})}}w}}.}}}}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

where w=(α, β, δ) are unknown feature coefficients to be estimated bylogistic regression, and lit(q_(ij)) are known prior log-odds that couldhave been derived from a different model. For instance, a uniform priorwould assume q_(ij)={circumflex over (p)}, where {circumflex over (p)}is the average CTR on the entire training corpus. Another possibility isto derive prior log-odds q_(ij) by combining relevance scores with clickfeedback.

To add new (page,ad) region combination, the system 100 only needs toaugment Equation 5 with the appropriate linear terms for the page mainand ad main effects. For the interaction effects, the system 100re-parameterizes its model to facilitate indexing. There-parameterization is explained here. The connection to indexing isdiscussed below in the Ad Search Prototype section. For each (page,ad)combination (r₁, r₂), a word w that occurs in both r₁ and r₂ has acoefficient δ_(w,r) ₁ _(,r) ₂ which depends on the word w, the pageregion and the ad region. We assume parameterization, as in

δ_(w,r) ₁ ,r ₂ =δ_(w)·γ_(p(r) ₁ ₎·γ_(a(r) ₂ ₎.  Equation 6.

In other words, the interaction of a word for a given page and ad regioncombination is factored into word-specific, page-specific andad-specific components. Accordingly, for M words, R₁ page regions, R₂ adregions, the number of parameters equals M+R₁+R₂ as opposed to M·R₁·R₂in the original model. The estimate of coefficients is obtained bymaximizing the log-likelihood of the data as given by

$\begin{matrix}{\sum\limits_{ij}\; {\left( {{y_{ij}{\log \left( p_{ij} \right)}} + {\left( {1 - y_{ij}} \right){\log \left( {1 - p_{ij}} \right)}}} \right).}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

where p_(ij) is given by Equation 5. The optimization problem describedabove may become ill-conditioned and lead to high variance estimates iffeatures tend to be correlated or are sparse or both. This is a drawbackin our scenario where feature sparsity and correlations are routine. Toprovide a robust solution, the system 100 put additional constraints onthe coefficients in the form of priors.

A prior of N(0, σ²) would mean that the parameter estimates are pinneddown in the range (−3σ, 3σ) with 99% probability a-priori. In theabsence of enough information about the coefficient from data, thisensures that the coefficient estimates do not diverge to the boundariesand cause numerical instability. To put more stringent constraints onsparse features, the system 100 down-weights the prior variance σ² by ameasure of relative sparsity, which is the variance of the featureoccurrence process relative to average feature occurrence variance. Thefeature occurrence variance is given by s(1−s), where s is the fractionof times the feature occurs. In particular, a set of relationships isprovided as

$\begin{matrix}{\begin{matrix}{\alpha_{w} \sim {N\left( {0,{\sigma^{2} \cdot \frac{{s_{p}(w)}\left( {1 - {s_{p}(w)}} \right)}{s_{p}\left( {1 - s_{p}} \right)}}} \right.}} \\{\beta_{w} \sim {N\left( {0,{\sigma^{2} \cdot \frac{{s_{a}(w)}\left( {1 - {s_{a}(w)}} \right)}{s_{a}\left( {1 - s_{a}} \right)}}} \right.}} \\{\delta_{w} \sim {N\left( {0,{\sigma^{2} \cdot \frac{{s_{I}(w)}\left( {1 - {s_{I}(w)}} \right)}{s_{I}\left( {1 - s_{I}} \right)}}} \right.}}\end{matrix}.} & {{Equation}\mspace{14mu} 8}\end{matrix}$

Note that separate averages are used for the main page and ad effects,and interaction effects (indicated by the subscripts p, a, and I). Inreal experiments, σ²=9; experiments with several other values in therange of 3 to 20 have been found not to yield much difference in theresults.

Now, the optimization problem reduces to estimating the coefficients bymaximizing the log-posterior which is the sum of the log-likelihood(Equation 7) and the log-prior of the coefficients, as discussed above.Next, the optimization process itself is discussed.

Several approaches to optimize the objective function exist. Among theones that have been used in large-scale applications are iterativescaling, nonlinear conjugate gradient, quasi-Newton,iteratively-reweighted least squares, truncated Newton, and trust-regionNewton. All these methods are iterative and generate a sequence ofestimates that converge to the optimal solution. For all methods exceptiterative scaling, cost per iteration is high but the convergence isfast. For iterative scaling which updates one component at a time, costper iteration is low but convergence is slower. For application here,the training corpus typically has several million data points andseveral thousand features, making it extremely slow to fit the modelusing these approaches on a single machine. To scale the computations,the system 100 utilizes a simple parallelization approach that randomlysplits the data into several parts, fits a logistic regressionseparately to each part and then combines the estimates obtained fromeach piece. For convenience, the system 100 may perform its computationin a MapReduce framework. MapReduce is a conventional programming modelfor processing large data sets. It runs on a large cluster of commoditymachines; it is highly scalable processing several gigabytes of data onthousands of machines and easy to use. The run-time system automaticallytakes care of the details of partitioning the data, scheduling jobsacross machines, handling failures and managing inter-machinecommunication.

To fit a logistic regression for a given piece, the system 100 uses asimple iterative scaling (also known as conditional maximization)approach. The algorithm is as follows: Initialize the coefficients α's,β's, and δ's to 0, and γ_(p( ))'s and γ_(a( ))'S to 1; then update thevalue of each coefficient one at a time holding the others fixed at thecurrent value by maximizing the likelihood through a Newton-Raphsonmethod. This completes a single iteration. The procedure is continuedthrough several iterations until convergence. The method issubstantially guaranteed to converge since every step can only increasethe likelihood. Along with a coefficient estimate, the Newton-Raphsonprocedure provides an estimate of the negative Hessian, the inverse ofwhich provides an estimate of variance of the coefficient from maximumlikelihood theory. The results on the various data partitions arecombined using a weighted average of the individual estimates, where theweight assigned to partition-specific estimate is its relative precisionobtained from the negative Hessian values. This weighting scheme is thesubstantially best way to combine estimates through a linear function.

Ad Search Prototype

FIG. 2 is a schematic diagram of a system 200 for providing better webad matching by combining relevance with consumer click feedback, inaccordance with an embodiment of the present invention. A key feature ofthe system 200 is that it is suitable for efficient evaluation overinverted ad indexes. This section discusses an implementation of thesystem 200, which is a prototype ad search engine based on a query pageevaluation algorithm and inverted indexing of the ads. The relevancedevice 124 including the page evaluation device 130 may involve, forexample, calculations using a conventional WAND algorithm. The clickfeedback device 114 including the ad inversion sort device 120 mayinvolve, for example, calculations using a conventional Hadoop computingframework.

The system 200 allows for any kind of feature to be used in the adsearch. The system 200 uses unigrams, phrases and classes as features.The ads index database 122 (i.e., inverted index) is composed of onepostings list for each feature that has one entry (i.e., posting) foreach ad that contains this feature. The ads are represented in the adsindex database 122 by adIDs, which are unique numeric identifiersassigned to each ad.

Consumers 108 from a multiple consumer computers 106 click on ads on webpages. The front end server 104 informs the ads click feedback device114 of the ads clicked. The click feedback device 114 also has access tothe ads database 112.

The system 200 produces the inverted ad index over preferably a grid ofmachines running the ad inversion framework. The indexing starts withthe ad feature extraction device 116 extracting features from the ads.The ad identification assignment device 118 represents each feature by aunique numeric featureID and sorts the resulting data file by <adID,featureId>. Next, the ad inversion sort device 120 inverts this file bysorting the file by <featureId, adID> as a key. The ad indexing device121 then writes the inverted data file (delta compressed posting lists)into the ads index database 122. The system 200 uses the ads indexdatabase 122 later during query runtime in order to evaluate queries.

There are a few important differences in this ad search engine thatrequire a different approach compared to web search engines. First, inweb search, the queries are short and the documents are long. In thepresent ad search case, the number of features per ad is usually lowerthan the number of features extracted from a web page, which representthe ad space query here. So, it is almost never the case that an ad willcontain all the features of the ad search query. Accordingly, the adsearch engine performs similarity search in the vector space with a longquery and relatively short ad vectors. In contrast, for the majority ofthe web queries, there are many pages that contain all the query words.One of the key issues is how to rank the pages containing the query.

The relevance device 124 includes architecture for analyzing content ofquery pages where the ads are shown. The page feature extraction device126 extracts features from a query page. The page feature re-weightingdevice 128 breaks down the query page into a bag of pairs <featureId,weight>. For each query page feature, the page evaluation device 130opens a cursor over the posting list of this feature. During theevaluation, the page evaluation device 130 moves the cursors forwardexamining the documents (ads) as the documents are encountered from theads index database 122. The page evaluation device 130 is configured tofind the next cursor to be moved based on an upper bound of the scorefor the documents at which the cursors are currently positioned. Thepage evaluation device 130 keeps a heap of current candidates. Theinvariant of the page evaluation device 130 is that the heap containsthe substantially best matches (highest scores) among the documents(ads) with IDs less than the document pointed by the current minimumcursor.

Cursors pointing on documents with upper bound smaller than the minimumscore among the candidate documents are candidates for a move. To findthe upper bound for a document, the page evaluation device 130 assumesthat all cursors that are before the current will hit this document(i.e. the document contains all those terms represented by cursorsbefore or at that document). It has been shown that the system 200 canuse the page evaluation device 130 with any function that is monotonicwith respect to the number of matching terms in the document. It canalso be easily shown that some non-monotonic scoring functions can alsobe used as long as the system 200 can find a mechanism to estimate thescore upper bounds.

One family of such functions is a set of functions where a fixed subsetof the features (known a priori) always decreases the score. In suchcases, the upper bound estimates just assume that these features do notappear in the ad. An example of such function is a cosine similaritywhere some of the query page coefficients are negative. The scoringfunction proposed in this invention might have such coefficients andfits well within the framework of the page evaluation device 130.

The system 200 incorporates the logistic-regression based model, whichis an important feature of the present invention. The system 200modifies the scoring Equation 5 to exclude the page effect and usesEquation 5 as a scoring formula for the page evaluation device 130(e.g., WAND). The click feedback device 114 uses M_(a(r)) of Equation 2during indexing (i.e., sorting) to calculate a static score for eachindividual ad. The ad identification assignment device 118 uses thisscore to assign an adID to the ads in decreasing ad score order. Thisscoring allows for estimating upper bounds of the ads that are skippedby using the score of the ad pointed by the preceding cursor in thesorted cursor list. The ad indexing device then writes the indexed adsto the ads index database 122.

After the relevance device 124 parses the page and extracts the featuresalong with their tf-idf scores, the page re-weighting device 128 appliesthe reweighing based on the I_(w) of Equation 4. The click feedbackdevice does not use M_(p(r)) of Equation 1 in the ad selection. Rather,the click probability calculation device 132 may use M_(p(r)) ofEquation 1 to adjust the final scores to calculate the probabilitiesaccording to Equation 5.

Method Outline

FIG. 3 is a flowchart of a method 300 of indexing ads in order toprovide better web ad matching, in accordance with an embodiment of thepresent invention. The method 300 starts in step 302 where the systemreceives ads that were clicked at a consumer computer. The clickfeedback device 114 of FIG. 2 may be configured to carry out this step302. The method 300 then moves to step 304 where the system extracts adfeatures from the ads. The ad feature extraction device 116 of FIG. 2may be configured to carry out this step 304. Next, in step 306, thesystem represents each feature by a unique featureID and sorts theresults data file by <adID, featureId>. The ad identification assignmentdevice 118 of FIG. 2 may be configured to carry out this step 306. Themethod 300 then proceeds to step 308 where the system inverts the datafile to sort the data file by <featureId, adID> as a key. The adinversion sort device 120 of FIG. 2 may be configured to carry out thisstep 308. Then, in step 310, the system writes the inverted data fileinto an ads index database. The ad indexing device 121 of FIG. 2 may beconfigured to carry out this step 310. The sorting and indexing in themethod 300 include use of logistic regression according to M_(a(r)) ofEquation 2. The method 300 is then at an end.

FIG. 4 is a flowchart of a method 400 for comparing query pages toindexed ads in order to provide better web ad matching, in accordancewith an embodiment of the present invention. The method 400 starts instep 402 where the system receives a query page from a consumercomputer. The relevance device 124 of FIG. 2 may be configured to carryout this step 402. The method 400 then moves to step 404 where thesystem extracts features from the query page. The page featureextraction device 126 of FIG. 2 may be configured to carry out this step404. Next, in step 406, the system re-weights the query page. The pagefeature re-weighting device 128 of FIG. 2 may be configured to carry outthis step 128. The re-weighting is based on I_(w) of Equation 4. Themethod 400 then proceeds to step 408 where the system evaluates thequery page in light of each ad in order to score each ad and picksubstantially best ad matches from the ads index database written in themethod 300 of FIG. 3. The page evaluation device 130 of FIG. 2 may beconfigured to carry out this step 408. Note that the system computes ascore for almost all (page, ad) pair and then uses this score to judgewhich ads are the best for the given page. Substantially, the only timethe system does not compute such scores is when the system evaluatesthat the score will not be high enough for a particular ad to be amongthe substantially best ads. It is not the case that the system computesscores only for the substantially best ad matches. Then, in step 410,the system returns the substantially best ad match(es) to the consumercomputer. The relevance device 124 of FIG. 2 may be configured to carryout this step 410. The method 400 is then at an end.

The method 400 may involve an optional step where the system calculatesa final score for each of the substantially best ad matches. The clickprobability calculation device 132 of FIG. 2 may be configured to carryout this optional step. This final static scoring involves use ofM_(p(r)) of Equation 1 to adjust the final scores to calculate theprobabilities according to Equation 5.

The top scoring ads in the top ads database may be used later duringruntime of a query. Thus, better matching ads can be had for the query.

Computer Readable Medium Implementation

Portions of the present invention may be conveniently implemented usinga conventional general purpose or a specialized digital computer ormicroprocessor programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of application-specific integratedcircuits or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in theart.

The present invention includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to control, or cause, a computer to perform any of the processesof the present invention. The storage medium can include, but is notlimited to, any type of disk including floppy disks, mini disks (MD's),optical disks, DVDs, CD-ROMs, micro-drives, and magneto-optical disks,ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices(including flash cards), magnetic or optical cards, nanosystems(including molecular memory ICs), RAID devices, remote datastorage/archive/warehousing, or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human consumer orother mechanism utilizing the results of the present invention. Suchsoftware may include, but is not limited to, device drivers, operatingsystems, and consumer applications. Ultimately, such computer readablemedia further includes software for performing the present invention, asdescribed above.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the present invention, including without limitationreceiving a query page, extracting features from the query page,re-weighting the query page, evaluating the query page to obtainsubstantially best ad matches of the indexed ads, and calculating afinal score for each ad in order to pick substantially best ad matches,according to processes of the present invention.

Advantages

The system of the present invention provides a new model to combinerelevance with click feedback for a contextual advertising system. Themodel is based on a logistic regression and allows for a large number ofgranular features. The key feature of the modeling approach is theability to model interactions that exist among words between page and adregions in a way that is suitable for efficient evaluation over invertedindexes. In fact, the system employs a multiplicative factorization tomodel the interaction effects for several (page, ad) regions in aparsimonious way that facilitates fast look-up of ads at query runtime.Large scale experiments have been proven the advantage of combiningrelevance with click feedback. In fact, experiments have achieved a 25%lift in precision for a recall value of 10% relative to a pure relevancebased model.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

1. A method of comparing query pages to indexed ads in order to providebetter web ad matching, the method comprising: receiving a query page;extracting features from the query page; re-weighting the query page;evaluating the query page in light of each ad in order to score each adand pick substantially best ad matches of the indexed ads; and returningthe substantially best ad matches to the consumer computer.
 2. Themethod of claim 1, wherein the re-weighting the query page includesranking words of the query page based on a measure that quantifiesinteraction between words occurring in the query page and ad regions ofthe indexed ads.
 3. The method of claim 1, wherein the re-weighting thequery page includes ranking words of the query page by computing averagetf-idf scores across the query page and indexed ads for respectiveregions under consideration.
 4. The method of claim 1, furthercomprising calculating a final score for each ad in order to picksubstantially best ad matches, wherein the calculating the final scoreincludes using logistic regression to adjust the final score, whereinthe final score represents a click probability of a particular ad for aparticular query of the query page.
 5. A method of indexing ads in orderto provide better web ad matching, the method comprising: receiving adsthat were clicked at a consumer computer; extracting ad features fromthe ads; sorting the ads according to ad identification to provide adata file; and inverting the data file to sort the data file accordingto feature identification, wherein sorting the ads includes computing astatic score for each ad using parameters learnt from using logisticregression on some training data.
 6. The method of claim 5, wherein thelogistic regression is performed on regions of each ad, and wherein eachregion has a different impact on the static score.
 7. The method ofclaim 5, wherein the static score is used to assign ad identification tothe ads in decreasing ad score order.
 8. The method of claim 5, furthercomprising writing the inverted data to an ads index database.
 9. Themethod of claim 5, further comprising applying a variable selectiontechnique to select a subset of important words of the ad features to beused in the logistic regression.
 10. The method of claim 9, wherein thevariable selection technique includes at least one of: using clicks andviews on the ads; and using relevance scores of words of the ads thatare independent of click feedback.
 11. An apparatus for comparing querypages to indexed ads in order to provide better web ad matching, whereinthe apparatus is configured to receive a query page, the apparatuscomprising: a page feature extraction device configured to extractfeatures from the query page; a page feature re-weighting deviceconfigured to re-weight the query page; a page evaluation deviceconfigured to evaluate the query page in light of each ad in order toscore each ad and pick substantially best ad matches of the indexed ads,wherein the apparatus is configured to return the substantially best admatches to the consumer computer.
 12. The apparatus of claim 11, whereinthe page feature re-weighting device is further configured to rank wordsof the query page based on a measure that quantifies interaction betweenwords occurring in the query page and ad regions of the indexed ads. 13.The apparatus of claim 11, wherein the page feature re-weighting deviceis further configured to rank words of the query page by computingaverage tf-idf scores across the query page and indexed ads forrespective regions under consideration.
 14. The apparatus of claim 11,wherein the click probability calculation device is configured tocalculate a final score for each ad in order to pick substantially bestad matches and to use logistic regression to adjust the final score,wherein the final score represents a click probability of a particularad.
 15. An apparatus for indexing ads in order to provide better web admatching, wherein the apparatus is configured to receive ads that wereclicked at a consumer computer, the apparatus comprising: an ad featureextraction device configured to extract ad features from the ads; an adidentification assignment device configured to sort the ads according toad identification to provide a data file; and an ad inversion sortdevice configured to invert the data file to sort the data fileaccording to feature identification, wherein the apparatus is furtherconfigured to sort the ads by computing a static score for each ad usingparameters learnt from using logistic regression on some training data.16. The apparatus of claim 15, wherein the apparatus is furtherconfigured to perform logistic regression on regions of each ad, andwherein each region has a different impact on the static score.
 17. Theapparatus of claim 15, wherein the apparatus is further configured touse the static score to assign ad identification to the ads indecreasing ad score order.
 18. The apparatus of claim 15, furthercomprising an ad indexing device configured to write the inverted datato an ads index database.
 19. The apparatus of claim 15, wherein theapparatus is further configured to apply a variable selection techniqueto select a subset of important words of the ad features to be used inthe logistic regression.
 20. The apparatus of claim 19, wherein thevariable selection technique includes at least one of: using clicks andviews on the ads; and using relevance scores of words of the ads thatare independent of click feedback.
 21. A computer readable mediumcarrying one or more instructions for comparing query pages to indexedads in order to provide better web ad matching, wherein the one or moreinstructions, when executed by one or more processors, cause the one ormore processors to perform the steps of: receiving a query page;extracting features from the query page; re-weighting the query page;evaluating the query page in light of each ad in order to score each adand pick substantially best ad matches of the indexed ads; and returningthe substantially best ad matches to the consumer computer.